Elamur Research⌕
☰
home / papers / 2019 / Do Massively Pretrained Language Models Make Better Storytellers?

Do Massively Pretrained Language Models Make Better Storytellers?

Paper · 2019 · Abigail See, Aneesh Pappu, Rohun Saxena et al.

Summary

The paper asks whether large-scale open-domain pretraining actually improves neural text generation, using story writing as the test case. The authors finetune GPT2-117, the smallest publicly released GPT2 variant, on a filtered version of the WritingPrompts corpus. They compare it with the Fusion Model of Fan et al., a convolutional seq2seq system designed specifically for story generation. Instead of collecting human judgments, they vary top-k sampling from k = 1 up to the full vocabulary. They then score 150-word stories with many automatic metrics, covering prompt relatedness, lexical and syntactic diversity, repetition and part-of-speech properties. GPT2-117 ties its stories much more closely to the prompt, is more sensitive to event order and uses rarer words. At small k, however, both systems are equally repetitive and bland. At large k, both come close to human statistics. The authors conclude that these familiar failures stem mainly from likelihood-oriented decoding, not from model training or data size. They argue that the real open problems are coherence, reasoning and world knowledge.

Key points

  • Data: WritingPrompts contains 303,358 prompt–story pairs from r/WritingPrompts. The authors kept only pairs whose combined length fits GPT2's 1024-token BPE context, producing WritingPrompts-1024 with 192,364 train, 11,115 validation and 10,686 test examples.
  • Perplexity: Finetuned GPT2-117 (117M parameters) reaches a test word-level perplexity of 31.54, compared with 37.54 for the Fusion Model (255.4M parameters). Both are normalized per word token so the different tokenizations can be compared.
  • Prompt ranking accuracy: In this test, the true prompt competes against nine random prompts. GPT2-117 scores 80.16%, the Fusion Model 39.8%, and random chance 10%. GPT2-117 also copies more n-grams from the prompt.
  • Low-k vs. high-k text: With low k, both models produce more verbs and pronouns, fewer nouns and adjectives, narrower syntactic patterns and more context copying. With high k, the metrics match human text, but the stories lack coherence and commonsense.
  • Limitations and resources: The study uses only the smallest GPT2 and includes no human evaluation. The generated stories and evaluation code are released at github.com/abisee/story-generation-eval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,053 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola and Christopher D. Manning. 2019. Do Massively Pretrained Language Models Make Better Storytellers?. arXiv:1909.10705.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.10705 · arXiv 1909.10705

@misc{see2019do,
  title = {{Do Massively Pretrained Language Models Make Better Storytellers?}},
  author = {Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola and Christopher D. Manning},
  year = {2019},
  doi = {10.48550/arxiv.1909.10705},
  eprint = {1909.10705},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.10705}
}

References

37 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shlomo Argamon, Moshe Koppel, and Galit Avneri. 1998. Routing documents according to style. In First International workshop on innovative information systems. find
  2. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations. find
  3. Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. 2018. Generating more interesting responses in neural conversation models with distributional constraints. In Empirical Methods in Natural Language Processing. find
  4. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34.
  5. Scott F Beers and William E Nagy. 2009. Syntactic complexity as a predictor of adolescent writing quality: Which measures? which genre? Reading and Writing, 22(2):185–200. find
  6. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior Research Methods, 46(3):904–911. find
  7. Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2018. Language gans falling short. In NeurIPS Workshop on Critiquing and Correcting Trends in Machine Learning. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  9. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Association for Computational Linguistics.
  10. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning.
  11. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043.
  12. Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  13. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
  14. Molly E Ireland and James W Pennebaker. 2010. Language style matching in writing: Synchrony in essays, correspondence, and poetry. Journal of personality and social psychology, 99(3):549. find
  15. Shaojie Jiang and Maarten de Rijke. 2018. Why are sequence-to-sequence models so dull? Understanding the low-diversity problem of chatbots. In EMNLP Workshop on Search-Oriented Conversational AI. find
  16. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, Fog count and Flesch reading ease formula) for navy enlisted personnel. find
  17. Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. 2018. Importance of a search strategy in neural dialogue modelling. arXiv preprint arXiv:1811.00907. — [arXiv:1811.00907](https://arxiv.org/abs/1811.00907) find
  18. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  19. Jiwei Li and Dan Jurafsky. 2016. Mutual information and diverse decoding improve neural machine translation. arXiv preprint arXiv:1601.00372. find
  20. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Empirical Methods in Natural Language Processing.
  21. Huanru Henry Mao, Bodhisattwa Prasad Majumder, Julian McAuley, and Garrison W. Cottrell. 2019. Improving neural story generation by targeted common sense grounding. In Empirical Methods in Natural Language Processing. find
  22. Danielle S McNamara, Scott A Crossley, and Philip M McCarthy. 2010. Linguistic features of writing quality. Written communication, 27(1):57–86. find
  23. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Empirical Methods in Natural Language Processing.
  24. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. Fairseq: A fast, extensible toolkit for sequence modeling. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  25. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing.
  26. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  27. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI tech report. find
  28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI tech report. find
  29. Melissa Roemmele, Andrew S Gordon, and Reid Swanson. 2017. Evaluating story generation systems using automated linguistic analyses. In KDD Workshop on Machine Learning for Creativity. find
  30. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  31. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI Conference on Artificial Intelligence.
  32. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Association for Computational Linguistics.
  33. Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. 2018. Cold fusion: Training seq2seq models together with language models. In Proc. find
  34. Interspeech 2018, pages 387–391. find
  35. Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search: Decoding diverse solutions from neural sequence models. In AAAI Conference on Artificial Intelligence.
  36. Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Jun Xu, and Xueqi Cheng. 2018. Learning to control the specificity in neural response generation. In Association for Computational Linguistics. find
  37. Zachary M Ziegler, Luke Melas-Kyriazi, Sebastian Gehrmann, and Alexander M Rush. 2019. Encoder-agnostic adaptation for conditional language generation. arXiv preprint arXiv:1908.06938. — [arXiv:1908.06938](https://arxiv.org/abs/1908.06938) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/see-2019-massively-pretrained-language-models-make/