Elamur Research⌕
☰
home / papers / 2024 / Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers

Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers

Paper · 2024 · Melanie Subbiah, Sean Zhang, Lydia B. Chilton et al.

Summary

Testing how well LLMs grasp fiction is hard. Published stories have probably been seen during training, and judging long-form summaries is slow and expensive. This paper avoids both problems by working with skilled creative writers, who contributed their own unpublished short stories and then rated model summaries of them. The authors are the people best placed to spot misreadings. GPT-4, Claude-2.1 and Llama-2-70B-chat each summarized all 25 stories. Writers judged the summaries on coherence, faithfulness, coverage of key details and quality of interpretation, and the team added span-level error analysis informed by narrative theory. The two commercial models sometimes produced strong summaries with real thematic insight, and they handled longer stories about as well as shorter ones. Still, every model made faithfulness errors in more than half of its outputs. The models were often vague and misjudged subtext, characters' emotions and unreliable narrators. LLM-as-judge setups and standard automatic metrics tracked the writers' ratings only weakly. This suggests expert human evaluation remains necessary for nuanced narrative tasks.

Key points

  • Data: 9 writers (8 with a writing degree, 7 previously published) contributed 25 unpublished stories. The stories run up to about 9,900 tokens and are grouped into short, medium and long buckets of 10, 9 and 6 stories.
  • Novelty check: Llama-2-7B gave the stories perplexities of 9.2–21.6. By comparison, the first chapter of Pride and Prejudice scored about 1.3, which supports that the stories were unseen by the models.
  • Setup: GPT-4 and Claude-2.1 read whole stories, while Llama-2-70B used hierarchical chunk-then-merge summarization for longer texts. Claude declined to summarize 2 stories because of its content rules.
  • Metrics: Automatic metrics correlated weakly with writer scores, with GPT-4 as judge reaching about r = 0.29 for faithfulness. FABLES gave nearly identical averages (~0.99) to all three models.
  • Efficiency and release: Writers reviewed a story's summaries in roughly 5–10 minutes, compared with 60–90 minutes for the researchers. Code, writer ratings and error annotations are available on GitHub (reading-subtext).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSummarization · 232 works

Cite

Cite as Melanie Subbiah, Sean Zhang, Lydia B. Chilton and Kathleen McKeown. 2024. Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00702

@article{subbiah2024reading,
  title = {{Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers}},
  author = {Melanie Subbiah, Sean Zhang, Lydia B. Chilton and Kathleen McKeown},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00702},
  url = {https://doi.org/10.1162/tacl_a_00702}
}

References

54 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nina Begus. 2023. Experimental narratives: A Comparison of human crowdsourced storytelling and AI storytelling. arXiv preprint arXiv: 2310.12902. — [arXiv:2310.12902](https://arxiv.org/abs/2310.12902) find
  2. Wayne C. Booth. 1983. The Rhetoric of Fiction. University of Chicago Press. https://doi.org/10.7208/chicago/9780226065595 .001.0001 — [doi:10.7208/chicago/9780226065595](https://doi.org/10.7208/chicago/9780226065595) ↗
  3. Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2024. Art or artifice? Large language models and the false promise of creativity. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–34. https://doi.org/10 .1145/3613904.364273… find
  4. Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brahman, and Smaranda Muresan. 2023. Creativity support in the age of large language models: An empirical study involving emerging writers. arXiv preprint arXiv:2309. 12570. find
  5. Tuhin Chakrabarty, Vishakh Padmakumar, and He He. 2022. Help me write a poem - instruction tuning as a vehicle for collaborative poetry writing’’. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6848–6863, Abu Dhabi, United Arab Emirates. Association… find
  6. Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. Booookscore: A systematic exploration of book-length summarization in the era of LLMs. In The Twelfth International Conference on Learning Representations. find
  7. Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. SummScreen: A dataset for abstractive screenplay summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 8602–8615. https://doi.org/10 .18653/v1/2022.acl-long.589
  8. Tobias Daudert. 2020. A web-based collaborative annotation and consolidation tool. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7053–7059. find
  9. Nicholas Deas, Jessica Grieser, Shana Kleiner, Desmond Patton, Elsbeth Turcan, and Kathleen McKeown. 2023. Evaluation of African American language bias in natural language generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6805–6824, Singapor… find
  10. Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFact-Eval: Improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p… — [doi:10.18653/v1/2022](https://doi.org/10.18653/v1/2022) find
  11. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summ-Eval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. https://doi.org/10.1162/tacl a 00373 — [doi:10.1162/tacl](https://doi.org/10.1162/tacl) ↗
  12. Gérard Genette. 1980. Narrative Discourse: An Essay in Method, volume 3. Cornell University Press. find
  13. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022a. FALTE: A toolkit for fine-grained annotation for long text evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 351–358, Abu Dhabi, UAE. Association for Computational… find
  14. Jessica A. Grieser. 2022. The Black Side of the River: Race, Language, and Belonging in Washington, DC. Georgetown University Press. find
  15. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa…
  16. David Herman. 2009. Basic Elements of Narrative. John Wiley & Sons. https://doi.org/10.1002/9781444305920 — [doi:10.1002/9781444305920](https://doi.org/10.1002/9781444305920) ↗
  17. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in Neural Information Processing Systems, 28.
  18. Chieh-Yang Huang, Sanjana Gautam, Shannon McClellan Brooks, Ya-Fang Lin, and Ting-Hao ‘Kenneth’ Huang. 2023. Inspo: Writing stories with a flock of AIs and humans. arXiv preprint arXiv:2311.16521. — [arXiv:2311.16521](https://arxiv.org/abs/2311.16521) find
  19. Daphne Ippolito, Ann Yuan, Andy Coenen, and Sehmon Burnam. 2022. Creative writing with an AI-powered writing assistant: Perspectives from professional writers. arXiv preprint arXiv:2211.05030. — [arXiv:2211.05030](https://arxiv.org/abs/2211.05030) find
  20. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022b. News summarization and evaluation in the era of GPT-3. arXiv preprint arXiv:2209. 12356. find
  21. Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Xiangji Huang. 2024. A comprehensive evaluation of large language models on benchmark biomedical text processing tasks. Computers in Biology and Medicine, page 108189. https://doi.org/10.1016/j.compbiomed.2024.108189, PubMed: 38447502 ↗
  22. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022c. SNaC: Coherence error detection for narrative summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 444–463, Abu Dhabi, United Arab Emirates. find
  23. Association Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. Fables: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261. — [arXiv:2404.01261](https://arxiv.org/abs/2404.01261) find
  24. Peter Kincaid, Robert P. Fishburne, Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. https://doi.org/10.21236/ADA006655 — [doi:10.21236/ada006655](https://doi.org/10.21236/ada006655) ↗
  25. Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguisti…
  26. Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. BOOKSUM: A collection of datasets for long-form narrative summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6536–6558, Abu Dhabi, United Arab Emirates. Associa… find
  27. Faisal Ladhak, Bryan Li, Yaser Al-Onaizan, and Kathleen McKeown. 2020. Exploring content selection in summarization of novel chapters. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5043–5054, Online. Association for Computational Linguistics. https… find
  28. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  29. Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2024. Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization. In Findings of the Association f… find
  30. Ridwan Mahbub, Ifrad Towhid Khan, Samiha Shafiq Anuva, Md Shihab Shahriar, Md Tahmid Rahman Laskar, and Sabbir Ahmed. 2023. Unveiling the essence of poetry: Introducing a comprehensive dataset and benchmark for poem summarization. In The 2023 Conference on Empirical Methods in Natural Language Proce… find
  31. Jean M. Mandler and Nancy S. Johnson. 1977. Remembrance of things parsed: Story structure and recall. Cognitive Psychology, 9(1):111–151. https://doi.org/10.1016 /0010-0285(77)90006-8 find
  32. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
  33. Beloved. 1987. New York: Vintage. find
  34. OpenAI. 2023. GPT-4 Technical Report.
  35. Vishakh Padmakumar and He He. 2024. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations. find
  36. Alison H. Paris and Scott G. Paris. 2003. Assessing narrative comprehension in young children. Reading Research Quarterly, 38(1):36–76. https://doi.org/10.1598/RRQ.38.1.3 ↗
  37. Andrew Piper, Richard Jean So, and David Bamman. 2021. Narrative theory for computational narrative understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 298–311. https://doi.org/10.18653/v1/2021.emnlp-main.26 ↗
  38. Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. arXiv preprint arXiv:2309.09558. — [arXiv:2309.09558](https://arxiv.org/abs/2309.09558) find
  39. Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In Proceedings of the 61st Annual Meeting of the Association fo…
  40. Liyan Tang, Philippe Laban, and Greg Durrett. 2024a. MiniCheck: Efficient fact-checking of LLMs on grounding documents. arXiv preprint arXiv:2404.10774.
  41. Liyan Tang, Igor Shalyminov, Amy Wing mei Wong, Jon Burnsky, Jake W. Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKeown. 2024b. To-fueval: Evaluating hallucinations of llms on topic-focused dialogue summarization. arXiv preprin… — [arXiv:2402.13249](https://arxiv.org/abs/2402.13249) find
  42. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  43. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. 2022. SQuALITY: Building a long-document summarization dataset the hard way. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1139–1156, Abu Dhabi, United Arab Emirates.… find
  44. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862.
  45. Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, and Yulan He. 2024. OpenToM: A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models. arXiv preprint arXiv:2402.06044. find
  46. Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. 2022. Fantastic questions and where to find them: FairytaleQA – an authenti…
  47. Catherine Yeh, Gonzalo Ramos, Rachel Ng, Andy Huntington, and Richard Banks. 2024. Ghost-Writer: Augmenting collaborative human-AI writing experiences through personalization and agency. arXiv preprint arXiv:2402.08855. — [arXiv:2402.08855](https://arxiv.org/abs/2402.08855) find
  48. Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story writing with large language models. In 27th International Conference on Intelligent User Interfaces, pages 841–852. https://doi.org/10.1145/3490099.3511105 — [doi:10.1145/3490099.3511105](https://doi.org/10.1145/3490099.3511105) ↗
  49. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association…
  50. Qinglin Zhang, Chong Deng, Jiaqing Liu, Hai Yu, Qian Chen, Wen Wang, Zhijie Yan, Jinglin Liu, Yi Ren, and Zhou Zhao. 2023. Mug: A general meeting understanding and generation benchmark. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5.… find
  51. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. find
  52. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57. https://doi.org/10.1162/tacl a 00632 ↗
  53. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2023–2038, Abu Dhabi, U…
  54. Wenjie Zhong, Jason Naradowsky, Hiroya Takamura, Ichiro Kobayashi, and Yusuke Miyao. 2023. Fiction-writing mode: An effective control for human-machine collaborative writing. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1744–17… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/subbiah-2024-reading-subtext-evaluating-large-language/