Elamur Research⌕
☰
home / papers / 2017 / SemEval-2017 Task 7: Detection and Interpretation of English Puns

SemEval-2017 Task 7: Detection and Interpretation of English Puns

Paper · 2017 · Tristan Miller, Christian Hempelmann and Iryna Gurevych

Summary

Puns break a core assumption of word sense disambiguation: that each word in context carries a single intended meaning. This paper reports SemEval-2017 Task 7, the first shared evaluation for computational pun processing. The organisers released two manually annotated English data sets of short, self-contained texts such as jokes and aphorisms. One contains homographic puns, where both meanings share a spelling. The other contains heterographic puns, which rely on similar-sounding words such as "propane"/"profane". Each context has at most one pun, and pun meanings are tagged with WordNet 3.1 sense keys.

The task has three successive subtasks:

  • Detection: decide whether a context contains a pun.
  • Location: identify the pun word.
  • Interpretation: assign both of its senses.

Ten teams took part. Detection proved fairly tractable, but only a few systems beat a trivial baseline on location. Interpretation stayed very hard, with most systems failing to surpass the baselines. The authors argue that knowledge-based semantic analysis may be needed. The freely available data sets provide a benchmark for intentional lexical ambiguity.

Key points

  • Homographic data set: 2250 contexts in total, with puns present in 1607 (71%) of them. For 1298 of these puns, both meanings exist in WordNet 3.1. Inter-annotator agreement (Krippendorff's α with a MASI variant) was 0.777.
  • Heterographic data set: 1780 contexts, of which 1271 contain a pun. For 1098 (86%) of these puns, both meanings are in WordNet. Pre-adjudication agreement was α = 0.838.
  • Detection: F-scores ranged from 0.5587 to 0.8440.
  • Location: a "last word" baseline reached F1 0.4704 (homographic) and 0.5704 (heterographic). The best location scores were Idiom Savant at 0.6631 (homographic) and UWaterloo at 0.7964 (heterographic).
  • Interpretation: the out-of-competition Miller & Gurevych (2015) system remained the best homographic interpreter (P 0.1975, F1 0.1603). Only BuzzSaw and Duluth beat the most-frequent-sense baseline (F1 0.1348). On heterographic puns, the top system, Idiom Savant, scored only F1 0.0771.

Interesting ideas

  • I1. Position-based baseline. Picking the last word of the text as the pun draws on findings that punchlines tend to come at the end. Such a cheap, linguistically grounded baseline can expose how little some complex models actually add, in many span-location tasks.
  • I2. Maximum-polysemy baseline. Choosing the word with the most senses across all parts of speech, with ties broken toward the end, is a simple lexicon-only heuristic. It could serve as a sanity check for any task about locating deliberate ambiguity.
  • I3. Order-free scoring for two senses. An interpretation counts as correct if each predicted sense list is a non-empty subset of a different gold list, regardless of order. This lets multi-label or multi-meaning annotations be scored without penalising arbitrary ordering or partial overlap with broad gold sets.
  • I4. Coverage-aware metrics. Systems could abstain on some contexts and were scored on coverage, precision and recall separately. This rewards knowing when not to answer, which is useful in any evaluation where confident partial output is valuable.
  • I5. Cheap disagreement resolution. Annotator disagreements were first resolved automatically by intersecting the two sense sets, and only the remaining cases went to a human adjudicator. This reduces adjudication effort in sense or label annotation projects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Tristan Miller, Christian Hempelmann and Iryna Gurevych. 2017. SemEval-2017 Task 7: Detection and Interpretation of English Puns. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2005

@inproceedings{miller2017semeval,
  title = {{SemEval-2017 Task 7: Detection and Interpretation of English Puns}},
  author = {Tristan Miller, Christian Hempelmann and Iryna Gurevych},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2005},
  url = {https://doi.org/10.18653/v1/s17-2005}
}

References

47 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Debra Aarons. 2017. Puns and tacit linguistic knowledge. In Salvatore Attardo, editor, The Routledge Handbook of Language and Humor, Routledge, New York, NY, Routledge Handbooks in Linguistics, pages 80–94. ↗
  2. Salvatore Attardo. 1994. Linguistic Theories of Humor, Mouton de Gruyter, Berlin, chapter 2: The Linear Organization of the Joke. https://doi.org/10. 1515/9783110219029. ↗
  3. Salvatore Attardo and Victor Raskin. 1991. Script theory revis(it)ed: Joke similarity and joke representation model. Humor: International Journal of Humor Research 4(3–4):293–348. https://doi.org/10. 1515/humr.1991.4.3-4.293. ↗
  4. Yehoshua Bar-Hillel. 1960. The present status of automatic translation of languages. In Franz L. Alt, editor, Advances in Computers, Academic Press, volume 1, pages 91–163. on Semantic Evaluation (SemEval-2017). pages 456– 459. ↗
  5. Kim Binsted and Graeme Ritchie. 1994. An implemented model of punning riddles. In Proceedings of the Twelfth National Conference on Artificial Intelligence: AAAI-94. pages 633–638. ↗
  6. Justine T. Kao, Roger Levy, and Noah D. Goodman. 2015. A computational model of linguistic humor in puns. Cognitive Science 40(5):1270–1285. https://doi.org/10.1111/cogs.12269. ↗
  7. Kim Binsted and Graeme Ritchie. 1997. Computational rules for generating punning riddles. Humor: International Journal of Humor Research 10(1):25– 76. https://doi.org/10.1515/humr.1997.10. 1.25. ↗
  8. Shigeto Kawahara. 2010. Papers on Japanese imperfect puns. Online collection of previously published journal and conference articles. http://user.keio.ac.jp/~kawahara/pdf/punbook.pdf. ↗
  9. Fred J. Damerau. 1964. A technique for computer detection and correction of spelling errors. Communications of the ACM 7(3):171–176. https://doi.org/10.1145/363958.363994. ↗
  10. Stefan Daniel Keller. 2009. The Development of Shakespeare’s Rhetoric: A Study of Nine Plays. Number 136 in Swiss Studies in English. Narr, Tübingen. find
  11. Klaus Krippendorff. 1980. Content Analysis: An Introduction to Its Methodology. Number 5 in The Sage CommText Series. Sage Publications, Beverly Hills, CA. find
  12. Samuel Doogan, Aniruddha Ghosh, Hanyang Chen, and Tony Veale. 2017. Idiom Savant at SemEval-2017 Task 7: Detection and interpretation of English puns. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 103– 108. our card
  13. Michael Lesk. 1986. Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from a ice cream cone. In Virginia De-Buys, editor, SIGDOC ’86: Proceedings of the 5th Annual International Conference on Systems Documentation. pages 24–26. https://doi.org/10. 1145/318723.318728. ↗
  14. Christian F. Hempelmann. 2003a. Paronomasic Puns: Target Recoverability Towards Automatic Generation. Ph.D. thesis, Purdue University, West Lafayette, IN. find
  15. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10(8):707–710. find
  16. Christian F. Hempelmann. 2003b. YPS – The Ynperfect Pun Selector for computational humor. In Proceedings of the CHI 2003 Workshop on Humor Modeling in the Interface. find
  17. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, Cambridge. ↗
  18. Christian F. Hempelmann. 2008. Computational humor: Beyond the pun? In Victor Raskin, editor, The Primer of Humor Research, Mouton de Gruyter, Berlin, number 8 in Humor Research, pages 333–360. https://doi.org/10.1515/ 9783110198492.333. ↗
  19. Rada Mihalcea, Carlo Strapparava, and Stephen Pulman. 2010. Computational models for incongruity detection in humour. In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing: 11th International Conference, CIC-Ling 2010. Springer, Berlin/Heidelberg, number 6008 in Theoretical Computer Science and General Issues, pages 364–374. https://doi.org/10. 1007/978-3-642-12116-6_30. ↗
  20. Christian F. Hempelmann and Tristan Miller. 2017. Puns: Taxonomy and phonology. In Salvatore Attardo, editor, The Routledge Handbook of Language and Humor, Routledge, New York, NY, Routledge Handbooks in Linguistics, pages 95–108. find
  21. Bryan Anthony Hong and Ethel Ong. 2009. Automatically extracting word relationships as templates for pun generation. In Computational Approaches to Linguistic Creativity: Proceedings of the Workshop. pages 24–31. find
  22. Elena Mikhalkova and Yuri Karyakin. 2017. PunFields at SemEval-2017 Task 7: Employing Roget’s Thesaurus in automatic pun recognition and interpretation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 425–430. Lluís-F. find
  23. Hurtado, Encarna Segarra, Ferran Pla, Pascual Andrés Carrasco Gómez, and José Ángel González. 2017. ELiRF-UPV at SemEval-2017 Task 7: Pun detection and interpretation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 439–442. find
  24. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey. San Francisco, CA, pages 303–308. https://doi.org/10.3115/1075671.1075742. ↗ our card
  25. Vijayasaradhi Indurthi and Subba Reddy Oota. 2017. Fermi at SemEval-2017 Task 7: Detection and interpretation of homographic puns in English language. In Proceedings of the 11th International Workshop Tristan Miller. 2016. Adjusting Sense Representations for Word Sense Disambiguation and Automatic Pun Interpretation. Dr.-Ing. thesis, Department of Computer Science, Technische Universität Darmstadt. the OSTH. Journal of Literary Theory 3(2):285–312. https://doi.org/10.1515/JLT.2009.016. ↗
  26. Tristan Miller and Iryna Gurevych. 2015. Automatic disambiguation of English puns. In The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing: Proceedings of the Conference. volume 1, pages 719–729. find
  27. Graeme D. Ritchie. 2005. Computational mechanisms for pun generation. In Graham Wilcock, Kristiina Jokinen, Chris Mellish, and Ehud Reiter, editors, Proceedings of the 10th European Workshop on Natural Language Generation (ENLG-05). pages 125– 132. find
  28. Tristan Miller and Mladen Turković. 2016. Towards the automatic detection and identification of English puns. European Journal of Humour Research 4(1):59–75. ↗
  29. John Morkes, Hadyn K. Kernal, and Clifford Nass. 1999. Effects of humor in task-oriented human–computer interaction and computer-mediated communication: A direct test of SRCT theory. Human–Computer Interaction 14(4):395–435. https://doi.org/10. 1207/S15327051HCI1404_2. ↗
  30. Dieke Oele and Kilian Evang. 2017. BuzzSaw at SemEval-2017 Task 7: Global vs. local context for interpreting and locating homographic English puns with sense embeddings. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 443–447. find
  31. Frankie Rubinstein. 1984. A Dictionary of Shakespeare’s Sexual Puns and Their Significance. Macmillan, London. find
  32. Thorsten Schröter. 2005. Shun the Pun, Rescue the Rhyme? The Dubbing and Subtitling of Language-play in Film. Ph.D. thesis, Karlstad University. find
  33. Özge Sevgili, Nima Ghotbi, and Selma Tekir. 2017. N-Hance at SemEval-2017 Task 7: A computational approach using word association for puns. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 435–438. find
  34. Julia M. Taylor and Lawrence J. Mazlack. 2004. Computationally recognizing wordplay in jokes. In Kenneth Forbus, Dedre Gentner, and Terry Regier, editors, Proceedings of the Twenty-sixth Annual Conference of the Cognitive Science Society. pages 1315– 1320. find
  35. Martha Palmer, Hwee Tou Ng, and Hoa Trang Dang. 2007. Evaluation of wsd systems. In Eneko Agirre and Philip Edmonds, editors, Word Sense Disambiguation: Algorithms and Applications, Springer, number 33 in Text, Speech, and Language Technology, chapter 4, pages 75–106. ↗
  36. Ankit Vadehra. 2017. UWAV at SemEval-2017 Task 7: Automated feature-based system for locating puns. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 448– 451. find
  37. Rebecca J. Passonneau. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In 5th Edition of the International Conference on Language Resources and Evaluation. pages 831–836. our card
  38. Olga Vechtomova. 2017. UWaterloo at SemEval-2017 Task 7: Locating the pun using syntactic characteristics and corpus-based metrics. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 420–424. find
  39. Ted Pedersen. 2017. Duluth at SemEval-2017 Task 7: Puns upon a midnight dreary, lexical semantics for the weak and weary. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 384–388. find
  40. Annalu Waller, Rolf Black, David A. O’Mara, Helen Pain, Graeme Ritchie, and Ruli Manurung. 2009. Evaluating the STANDUP pun generating software with children with cerebral palsy. ACM Transactions on Accessible Computing 1(3):1–27. https://doi.org/10.1145/1497302.1497306. ↗
  41. Aniket Pramanick and Dipankar Das. 2017. JU_CSE_NLP at SemEval-2017 Task 7: Employing rules to detect and interpret English puns. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 431–434. find
  42. Leopold Wurth. 1895. Das Wortspiel bei Shakspere. Wilhelm Braumüller, Vienna. find
  43. Victor Raskin. 1985. Semantic Mechanisms of Humor. Number 24 in Studies in Linguistics and Philosophy. Springer Netherlands. https://doi.org/10. 1007/978-94-009-6472-3. ↗
  44. Victor Raskin, editor. 2008. The Primer of Humor Research. Number 8 in Humor Research. Mouton de Gruyter, Berlin. https://doi.org/10.1515/ 9783110198492. ↗
  45. Yuhuan Xiu, Man Lan, and Yuanbin Wu. 2017. ECNU at SemEval-2017 Task 7: Using supervised and unsupervised methods to detect and locate English puns. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 452– 455. find
  46. Toshihiko Yokogawa. 2002. Japanese pun analyzer using articulation similarities. In Proceedings of the 2002 IEEE International Conference on Fuzzy Systems: FUZZ 2002. volume 2, pages 1114– 1119. https://doi.org/10.1109/FUZZ.2002. 1006660. ↗
  47. Victor Raskin, Christian F. Hempelmann, and Julia M. Taylor. 2009. How to understand and assess a theory: The evolution of SSTH into the GTVH and now into find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/miller-2017-semeval-2017-task-7-detection/