📄 Hierarchical Neural Story Generation

Summary

The paper addresses the automatic generation of long works of fiction. Such texts must be coherent and fluent, and must also stay on a given topic. The authors collected the WritingPrompts corpus from the Reddit forum r/WritingPrompts. It contains about 300K human-written stories, each tied to a short prompt. Generation proceeds on two levels: first, a convolutional language model (GCNN) produces a prompt, then a convolutional seq2seq model expands it into a full story. A standard seq2seq model often ignores the prompt and behaves like plain language modeling. To avoid this, the authors propose fusion: a second seq2seq model is trained on top of an already trained first one, and it learns what the first one failed to capture. To let the model take distant context into account, the decoder is augmented with a gated multi-scale self-attention mechanism, in which each head attends to the preceding text with a different downsampling step. Together, these components markedly improve both automatic metrics and human judgments. The work matters for two reasons: it provides an open resource, and it demonstrates a practical way to plan long text through an intermediate level.

Key points

• WritingPrompts corpus: 303,358 "prompt–story" pairs collected over three years via the Reddit API. Split: 272,600 for training, 15,620 for validation, 15,138 for test. A prompt averages 28.4 words, a story 734.5 words. Code and data are available in the fairseq repository.
• Vocabularies: 19,025 tokens for prompts and 104,960 for stories. Only words occurring more than 10 times were included. Story length is capped at 1000 words.
• Perplexity: self-attention and fusion together reduce it by about 9 points compared to the baseline Conv seq2seq. Fusion outperforms an ensemble despite having fewer parameters.
• Human evaluation (Amazon Mechanical Turk): hierarchical stories are preferred twice as often as stories from a strong model without a prompt. Thanks to fusion, evaluators match a story to its prompt 7% more accurately.
• Text originality: in 150-word generated stories, the longest common subsequence with the training set averages 8.9 words. For the baseline Conv seq2seq it is 10.2 words, while KNN copies all 150. Known weaknesses of the model: repetition, and the generated prompts come out formulaic.