Elamur Research⌕
☰
home / papers / 2024 / Are Large Language Models Capable of Generating Human-Level Narratives?

Are Large Language Models Capable of Generating Human-Level Narratives?

Paper · 2024 · Yufei Tian, Tenghao Huang, Miri Liu et al.

Summary

The paper asks whether LLM-written stories match human ones in structure, not just fluency. The authors propose a framework that examines narrative discourse at three levels: macro story arcs (seven Vonnegut-style types), meso turning points (five types following Papalampidi et al., 2019) and micro affect (arousal and valence scored with the NRC-VAD lexicon). They gathered synopses of less-famous recent English-language films from Wikipedia, masked identifying details, and had GPT-4 write a counterpart story for each. Experts and automatic tools then annotated both sets. Compared with human plots, GPT-4 stories are uniformly upbeat, show little tension, cluster around a few arc types, and reach major setbacks and climaxes too early. Two new comprehension benchmarks show that most tested LLMs also understand narrative structure worse than humans, though generation is the weaker skill. Finally, prompting models to reason explicitly about arcs or turning points produces more diverse, suspenseful and emotionally engaging stories. This gives a measurable basis for evaluating and improving machine storytelling.

Key points

  • Dataset: 819 human film synopses from the Wikipedia category of 2020s English-language films, paired with GPT-4 versions for 1,638 synopses in total. It is released at github.com/PlusLabNLP/Narrative-Discourse.
  • In GPT-4 output, the turning points TP4 (Major Setback) and TP5 (Climax) appear earlier in the timeline, which indicates poor pacing and weaker suspense.
  • The comprehension benchmarks cover story arc identification and turning point identification. The models evaluated were Gemini Pro, Claude 3 Opus, Llama3, GPT-3.5 and GPT-4, and most fell short of humans. Supplying arc information improves turning point detection, and the reverse also holds.
  • Discourse-aware prompting gave clear gains over vanilla prompting:
    • Arc awareness raised diversity by about 45%.
    • Turning point guidance raised suspense and engagement by about 40%.
  • Limitations: the generation analysis covers only GPT-4, and all data are in English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May and Nanyun Peng. 2024. Are Large Language Models Capable of Generating Human-Level Narratives?. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.978

@inproceedings{tian2024are,
  title = {{Are Large Language Models Capable of Generating Human-Level Narratives?}},
  author = {Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May and Nanyun Peng},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.978},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.978}
}

References

49 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. Llama 3 model card. find
  2. AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1. find
  3. Tino Balio. 2013. A major presence in all of the world’s important markets: The globalization of hollywood in the 1990s. In Contemporary Hollywood Cinema, pages 58–73. Routledge. find
  4. Nina Begus. 2023. Experimental narratives: A comparison of human crowdsourced storytelling and ai storytelling. arXiv preprint arXiv:2310.12902. — [arXiv:2310.12902](https://arxiv.org/abs/2310.12902) find
  5. Anneliese Brei, Chao Zhao, and Snigdha Chaturvedi. 2024. Returning to the start: Generating narratives with related endpoints. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), p… find
  6. Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2024. Art or artifice? large language models and the false promise of creativity. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–34. find
  7. Prafulla Kumar Choubey, Aaron Lee, Ruihong Huang, and Lu Wang. 2020. Discourse as a function of event: Profiling discourse structure in news articles around the main event. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find
  8. Eric Chu, Jonathan Dunn, Deb Roy, Geoffrey Sands, and Russell Stevens. 2017. Ai in storytelling: Machines as cocreators. McKinsey & Company Media & Entertainment. find
  9. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. find
  10. Anjalie Field, Gayatri Bhat, and Yulia Tsvetkov. 2019. Contextual affective analysis: A case study of people portrayals in online\# metoo stories. In Proceedings of the international AAAI conference on web and social media, volume 13, pages 158–169. find
  11. Gustav Freytag. 1894. Die technik des dramas. S. Hirzel. find
  12. Rujun Han, Hong Chen, Yufei Tian, and Nanyun Peng. 2022. Go back in time: Generating flashbacks in stories with event temporal prompts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1450–1470. find
  13. Fabrice Y Harel-Canada, Hanyu Zhou, Sreya Muppalla, Zeynep Senahan Yildiz, Miryung Kim, Amit Sahai, and Nanyun Peng. 2024. Measuring psychological depth in language models. In Proceedings of The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  14. Kimmo Härmä, Sirpa Kärkkäinen, and Eila Jeronen. 2021. The dramatic arc in the development of argumentation skills of upper secondary school students in geography education. Education Sciences, 11(11):734. find
  15. Qiyue Hu, Bin Liu, Mads Rosendahl Thomsen, Jianbo Gao, and Kristoffer L Nielbo. 2021. Dynamic evolution of sentiments in never let me go: Insights from multifractal theory and its implications for literary analysis. Digital Scholarship in the Humanities, 36(2):322–332. find
  16. Tenghao Huang, Faeze Brahman, Vered Shwartz, and Snigdha Chaturvedi. 2021. Uncovering implicit gender bias in narratives through commonsense inference. ArXiv, abs/2109.06437. find
  17. Tenghao Huang, Ehsan Qasemi, Bangzheng Li, He Wang, Faeze Brahman, Muhao Chen, and Snigdha Chaturvedi. 2023. Affective and dynamic beam search for story generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11792–11806, Singapore. Association for Computational L… find
  18. Phyllis Kaniss. 1991. Making local news. University of Chicago Press. find
  19. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual difference…
  20. William Labov and Joshua Waletzky. 1967. Narrative analysis. In Essays on the Verbal and Visual Arts. University of Washington Press, Seattle, WA. find
  21. Susanne K Langer. 1942. Philosophy in a new key: A study in the symbolism of reason, rite, and art. Harvard University Press. find
  22. Boyang Li, Beth Cardier, Tong Wang, and Florian Metze. 2018. Annotating high-level structures of short stories and personal anecdotes. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  23. Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, and Ari Holtzman. 2024. Predicting vs. acting: A trade-off between world modeling & agent modeling. arXiv preprint arXiv:2407.02446. — [arXiv:2407.02446](https://arxiv.org/abs/2407.02446) find
  24. Yinhong Liu, Yixuan Su, Ehsan Shareghi, and Nigel Collier. 2024. Unlocking structure measuring: Introducing pdd, an automatic metric for positional discourse coherence. arXiv preprint arXiv:2402.10175. find
  25. Walaa Medhat, Ahmed Hassan, and Hoda Korashy. 2014. Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal, 5(4):1093–1113. find
  26. Annette Michelson. 1991. " where is your rupture?": Mass culture and the gesamtkunstwerk. October, 56:43–63. find
  27. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196.
  28. Saif Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174–184, Melbourne, Australia. Association for Computational Li…
  29. OpenAI. 2022. Introducing chatgpt. find
  30. OpenAI. 2023. Gpt-4 technical report.
  31. Pinelopi Papalampidi, Frank Keller, and Mirella Lapata. 2019. Movie plot analysis via turning point identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), page… find
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  33. Andrew J Reagan, Lewis Mitchell, Dilan Kiley, Christopher M Danforth, and Peter Sheridan Dodds. 2016. The emotional arcs of stories are dominated by six basic shapes. EPJ data science, 5(1):1–12. find
  34. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.…
  35. Maarten Sap, Anna Jafarpour, Yejin Choi, Noah A Smith, James W Pennebaker, and Eric Horvitz. 2022. Quantifying the narrative flow of imagined versus autobiographical stories. Proceedings of the National Academy of Sciences, 119(45):e2211715119. find
  36. Alexander Spangher, Kung-Hsiang Huang, Hyundong Cho, and Jonathan May. 2024a. Newsedits 2.0: Learning the intentions behind updating news. find
  37. Alexander Spangher, Jonathan May, Sz-Rung Shiang, and Lingjia Deng. 2021. Multitask semi-supervised learning for class-imbalanced discourse classification. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 498– 517. find
  38. Albin Zehe, Juliane Schröter, and Andreas Hotho. 2023. Towards a computational analysis of suspense: Detecting dangerous situations. ArXiv, abs/2305.06818. — [arXiv:2305.06818](https://arxiv.org/abs/2305.06818) find
  39. Alexander Spangher, Yao Ming, Xinyu Hua, and Nanyun Peng. 2022. Sequentially controlled text generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6848– 6866. find
  40. Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze. 2024b. Do llms plan like human writers? comparing journalist coverage of press releases with llms. In Proceedings of The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  41. Yufei Tian, Zeyu Pan, and Nanyun Peng. 2024. Detecting machine-generated long-form content with latent-space variables. Findings of The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  42. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  43. Teun A Van Dijk. 1980. Macrostructures: An interdisciplinary study of global structures in discourse, interaction, and cognition. Lawrence Erlbaum. find
  44. Kurt Vonnegut. 1995. Shapes of stories. Accessed June 2024. find
  45. Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. 2023. Learning personalized story evaluation. arXiv preprint arXiv:2310.03304. — [arXiv:2310.03304](https://arxiv.org/abs/2310.03304) find
  46. Winston Wu, Lu Wang, and Rada Mihalcea. 2023. Word category arcs in literature across languages and genres. In Proceedings of the The 5th Workshop on Narrative Understanding, pages 36–47. find
  47. Kaige Xie and Mark Riedl. 2024. Creating suspenseful stories: Iterative planning with large language models. In Conference of the European Chapter of the Association for Computational Linguistics. find
  48. Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4393–4479. find
  49. Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378–7385. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2024-are-large-language-models-capable/