📄 It's not Rocket Science: Interpreting Figurative Language in Narratives

Summary

Transformer-based language models handle literal text well but often fail on figurative expressions. Idioms and similes are hard because their meaning cannot be built from the meanings of their parts. The authors frame this as a story-continuation problem, following the Story Cloze setup. They took short passages from the Toronto Book corpus, each containing an idiom or a simile. Crowd workers then wrote continuations that make sense, or do not, depending on whether the expression is understood correctly. Two tasks are defined: choosing the plausible ending from two candidates, and generating one. Pre-trained models, GPT-3 included, perform poorly in zero-shot and few-shot settings. Supervised models still trail human performance. To narrow this gap, the authors add knowledge that mirrors two human interpretation strategies. ParaCOMET supplies inferences from the surrounding context, and COMET-ConceptNet supplies literal knowledge about the constituent words. These knowledge-enhanced models beat the baselines on both tasks, with the clearest gains in generation. The work offers a benchmark and evidence that cognitively motivated knowledge helps machines understand figurative language.

Key points

• Two datasets of fictional narratives, one for idioms and one for similes, drawn from the Toronto Book corpus. Each comes with crowdsourced plausible and implausible continuations. Code and data are on GitHub (FigurativeNarrativeBenchmark), with a leaderboard at AllenAI.
• Even supervised models fall short of humans. On the discriminative task, the accuracy gap is 10 points for idioms and 14.6 for similes. On the generative task, human-judged plausibility trails by about 24 and 28 points.
• The context model (ParaCOMET) works best for idioms, and the literal model (COMET-ConceptNet) works best for similes. This matches Cooper's (1999) findings on how people interpret idioms and how similes rely on attributes of the vehicle.
• In the absolute human evaluation, GPT2-XL continuations were rated plausible 56% (idioms) and 60% (similes) of the time. The best knowledge-enhanced models reached 68% and 76%, and human references reached 80% and 88%.
• Rouge-L and BERT-Score show little difference between the knowledge-enhanced models and the GPT-2 XL baseline, unlike the human judgments. Human-evaluation agreement was moderate (Krippendorff's α = 0.68 for the absolute evaluation, 0.63 for the comparative one).