📄 Prose for a Painting

Summary

The paper tackles generating Shakespeare-styled text about a painting. Ordinary image captions are short and literal, and no sizeable corpus pairs paintings with Shakespearean prose exists. The authors therefore chain two separately trained systems. First, an existing image-to-poem model produces a modern English poem. It uses an actor-critic design with object, sentiment and scene CNNs, skip-thought poetic clues, and a policy-gradient generator rewarded by two discriminators. Second, a seq2seq style-transfer model rewrites that poem into Shakespearean English. It is trained on line-aligned modern paraphrases of Shakespeare's plays, with word embeddings retrofitted using a Shakespearean-to-modern dictionary. Attention-based seq2seq with global attention beat a pointer-network variant on BLEU. A small human study rated the outputs moderately well for content, creativity and style. The work matters as a proof of concept: chaining models through an intermediate representation can make up for missing end-to-end parallel data. Code and models are released.

Key points

• Poem stage data: MultiM-Poem (image–poem pairs), UniM-Poem (large poem corpus), and MultiM-Poem(Ex), which adds each pair's three nearest neighbour poems from the corpus.
• Style-transfer data: modern paraphrases of 16 Shakespeare plays, split into 18,395 training, 1,218 validation and 1,462 test sentences.
• Average target BLEU was 29.65 for seq2seq with global attention, versus 26.97 with a pointer network. Global attention also outperformed Bahdanau attention.
• In the human evaluation, 32 students gave 1–5 Likert ratings: content 3.7, creativity 3.9, Shakespearean style 3.9. BLEU fell as source sentences grew longer.
• Setup: single-layer LSTM encoder and decoder, 192-dimensional retrofitted embeddings shared between source and target, Adam at learning rate 0.001, 25 epochs, TensorFlow 2.0 on Colab.

Interesting ideas

• I1. When no direct input→target-style data exists, route through an intermediate form that has its own data and keep style as a separate final step; each piece can be trained and checked on its own.
• I2. A small bilingual dictionary (archaic↔modern words) was used to retrofit pretrained embeddings, giving a cheap boost when parallel data is scarce.
• I3. Unknown output tokens are replaced by the source word that received the most attention, a simple copy fallback for names and rare words.
• I4. BLEU is reported as a function of source length rather than as one average, which shows where quality degrades.
• I5. Human ratings were split into separate axes (content relevance, creativity, style fidelity) instead of one overall score.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): Generation from the meaning graph could first produce neutral English, then apply a separate register/style pass, so the graph→text step stays explainable and style stays swappable.
• I2 (tokens): A curated archaic-form table (thou/thee/hath/doth) could feed the lemmatizer and morphology in crate en, so older fable and tale texts map onto modern lemmas.
• I3 (back): When generating text from the graph, rare names and entities should be copied verbatim from the source mention rather than regenerated, with the alignment logged as the explanation.
• I4 (learn): Parser and answer evaluations could be bucketed by sentence length so regressions on long sentences are not hidden by the average.
• I5 (answer): Free-text answers on tales could be judged separately for factual grounding in the text-world and for fluency, keeping correctness failures distinct from wording problems.