📄 BPoMP: The Benchmark of Poetic Minimal Pairs – Limericks, Rhyme, and Narrative Coherence

Summary

The authors test whether large pretrained language models capture rhyme and the basic narrative logic of a poem. To do so, they carry the BLiMP approach over to poetry and create the first task set of the BPoMP series. The material consists of English-language limericks: five-line poems with the rhyme scheme AABBA. The corpus was collected from the OEDILF website and filtered with automatic checks of rhyme and syllable count. From it, 10,000 minimal pairs of the form "original — corrupted variant" were built for each of three tasks. In the corrupted variant, either two rhyming line-final words are swapped, or two rhyming lines are swapped, or one rhyming word is replaced with a non-rhyming synonym. The model must assign a higher probability to the original. For BERT, pseudo-likelihood is used instead of ordinary probability. All Transformer models perform better than random guessing but fall noticeably short of humans. The largest gap appears where only the rhyme is lost. The work proposes poetry as a tool for evaluating language models and releases a curated corpus of limericks.

Key points

• Corpus: after basic filtering, 98,454 limericks remained from OEDILF (1624 authors). After checking rhyme with CMUdict and syllables with a character-level RNN, the clean subset retained 29,853 texts, i.e. 879,653 tokens and 52,316 unique words.

• Synonyms: for the third task, synonyms were taken from the Merriam-Webster Collegiate Thesaurus API and filtered by part of speech with spaCy. Among the candidates, the one closest to the original in its BERT LARGE contextual representation was chosen.

• Model accuracy:

  • rhyming word swap: GPT-2 0.9423, BERT 0.9414;
  • line swap: GPT-2 is best with 0.8336;
  • synonym replacement: BERT (cased) is best with 0.7034, GPT-2 scores 0.6557.

  Transformer-XL is consistently the weakest model; XLNet is close to the leaders.

• Humans: accuracy is 95–100%, 98.3% on average. However, the sample is small: 11 participants in one task and 2 in each of the others.

• GPT-2 error analysis: in the cases where the model is most confident, the log-probability difference is well explained by bigram frequency (correlation 0.71 and 0.89). If less confident cases are added, however, the correlation drops to −0.39.