📄 Idiom Savant at Semeval-2017 Task 7: Detection and Interpretation of English Puns

Summary

Idiom Savant is a University College Dublin system for SemEval-2017 Task 7. The task has three parts: deciding whether a short text contains a pun, locating the pun word, and naming the two WordNet senses it plays on. The authors treat the two pun types separately.

For heterographic (sound-alike) puns, they assume a pun bends a familiar phrase. The system swaps a word for a phonetically close one, using the CMU pronouncing dictionary and Levenshtein-based distances. It then checks in the Google n-gram corpus whether the new phrase is much more frequent than the original.

For homographic (same-spelling) puns, the system uses Word2Vec similarity. It compares the WordNet glosses of each candidate word with the surrounding context, looking for a word whose different senses are anchored by different parts of the sentence.

Results were solid on detection (about 0.84 F on both pun types) and moderate on location (about 0.66). Sense annotation was very weak (under 0.09). The work shows that cheap frequency and phonetic signals can capture much of pun detection, while sense interpretation needs richer context.

Key points

• Heterographic score: the gain in n-gram frequency after a single-word swap, minus a penalty for poor phonetic similarity. The penalty is the inverse similarity ratio raised to an exponent, tuned to 10. Example: "a jab well done" occurs 890 times, while "a job well done" occurs 203,575 times.
• Phonetic similarity: the maximum of three length-normalized ratios. One counts CMU phonemes as units, one compares concatenated phoneme strings, and one falls back to plain spelling for words missing from the dictionary. Lexical stress is dropped because the coarser representation worked better.
• Detection setup: trigrams were used for detection, with a threshold of −14. A baseline that flags texts containing a rare quadgram (frequency below 150) reached P 0.90, R 0.71, F 0.79. The system improved on it by roughly five F points.
• Tom Swifty puns: these adverb-final puns (e.g. "piously") got a dedicated path. It uses the NLTK POS tagger, a check for "-ly" endings, and phonetic and spelling prefix trees to recover the intended word.
• Homographic model:
  • It scores context–gloss similarity, with a POS damping factor of 0.2 when the two words share a tag.
  • It applies an "optimal innovation" correction that rewards moderate similarity, neither too high nor too low.
  • A text counts as a pun if its average score exceeds 0.6.
  • Results: F 0.84 on detection (all-positive baseline 0.83), 0.66 on location, 0.08 on senses.

Interesting ideas

• I1. Spot wordplay by rewriting one word into a sound-alike and measuring how sharply the phrase's corpus frequency jumps; a big jump signals a deliberately distorted idiom.
• I2. Combine several similarity measures (phoneme units, phoneme strings, raw spelling) and keep the best one, so each metric covers the others' blind spots and out-of-dictionary words still get a score.
• I3. Score a change rather than an absolute value: subtracting the original phrase's frequency means ordinary, already-common wording earns nothing.
• I4. Treat similarity as a curve rather than a monotone signal: links between a word and its context are most telling when they are moderate, neither near-identical nor unrelated.
• I5. Keep a simple, high-precision baseline (here, "contains a rare quadgram") beside the full system to show which part of the gain comes from the clever component.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): the irony/implicature layer could flag pun readings this way and record both the surface word and the recovered idiom word as competing frame fillers, keeping the reasoning explainable.
• I2 (tokens): misspelling and variant repair in the tokenizer could use the same max-of-metrics approach over CMU phonemes and characters before lemmatization.
• I3 (learn): self-induced repair rules could be accepted only when they raise a parse or plausibility score relative to the unrepaired version, not merely when the result looks frequent.
• I4 (frames): when choosing a VerbNet sense, an overly perfect match could be down-weighted in figurative contexts, so a second sense can stay alive as a possible double reading.
• I5 (learn): evaluations could routinely report such trivial-rule controls next to the parser or reader, making it visible when an improvement is only a frequency artifact.